Enhancing Alert Prioritization in Enterprise Security Operations via Factor Graph Modeling

Abstract

Advanced Persistent Threats (APTs) remain one of the most challenging problems in enterprise cybersecurity due to their stealthy, multi-stage nature and ability to evade traditional detection methods. While existing systems like RANK utilize factor graph-based scoring to correlate alerts into actionable incident graphs, they rely on static, manually defined tactic transition weights, limiting adaptability to emerging attack pat- terns. In this paper, we propose an enhancement to the RANK architecture by introducing an adaptive scoring mechanism based on pairwise factor graphs. Our approach dynamically models contextual and temporal relationships between MITRE ATT&CK tactics, enabling more accurate and flexible incident evaluation. Experiments conducted on two enterprise-scale datasets show that our method improves detection precision and F1 scores while maintaining compatibility with real-world alert streams. These results demonstrate the potential of adaptive, context- aware scoring in reducing false positives and enhancing threat detection capabilities in modern Security Operations Centers.

Introduction

Advanced Persistent Threats (APTs) are stealthy, multi-stage cyberattacks that evade traditional intrusion detection systems (IDS) due to their long duration and complexity. Existing IDS often produce excessive alerts, causing alert fatigue and missed threats. The AI-assisted system RANK improves detection by summarizing alerts into high-confidence incidents using a static factor graph model based on the MITRE ATT&CK framework. However, RANK’s fixed transition matrix limits its adaptability to evolving attacker behaviors.

This work proposes an extension to RANK by introducing an adaptive pairwise factor graph scoring mechanism that dynamically models inter-tactic relationships using contextual data like temporal proximity and historical co-occurrence. This adaptive scoring enhances precision, reduces false positives, and better handles variations in attack patterns without needing manual updates or retraining.

The method was tested on two enterprise-scale datasets and outperformed the original RANK system in precision and F1 score while maintaining recall. The adaptive model also demonstrated resilience against modified attack sequences. Limitations include reliance on heuristic alert partitioning and potential scalability concerns, which can be addressed in future work through end-to-end learning, online updates, graph neural networks, and improved explainability.

Conclusion

In this paper, we presented an extension to the RANK architecture for APT detection by introducing an adaptive scoring mechanism based on pairwise factor graphs. Unlike traditional approaches that rely on static transition matrices, our method captures contextual and temporal relationships between MITRE tactics, enabling more flexible and precise incident evaluation. We demonstrated that our approach achieves higher pre- cision and F1 scores compared to the original RANK sys- tem, while maintaining scalability and compatibility with enterprise-scale alert datasets. The use of dynamic edge weighting within the factor graph structure allows the system to adapt to evolving attacker behaviors without requiring manual updates or retraining. Our results show that incorporating adaptive scoring into incident graphs improves detection quality, reduces false pos- itives, and enhances interpretability. These characteristics are particularly valuable for security operations centers (SOCs) where timely and accurate threat triage is critical. Future work will focus on improving scalability, integrating learning-based representations, and expanding cross-domain generalizability. Overall, our approach contributes toward building more intelligent, context-aware, and actionable APT detection frameworks in enterprise environments.

References

[1] P. Chen, L. Desmet, and W. Joosen, “A study on advanced persistent threats,” IFIP International Conference on Communications and Multi- media Security, pp. 63–72, 2014. [2] N. Moustafa and J. Slay, “A survey of intrusion detection systems using machine and deep learning,” arXiv preprint arXiv:1904.03496, 2019. [3] MITRE Corporation, “MITRE ATT&CK Framework,” 2023, https:// attack.mitre.org/. [4] S. Garcia, A. Zunino, and M. Erquiaga, “An empirical study on alert fatigue in intrusion detection systems,” Computers & Security, vol. 88, [5] p. 101620, 2020. [6] H. M. Soliman, D. Sovilj, G. Salmon, M. Rao, and N. Mayya, “Rank: Ai-assisted end-to-end architecture for detecting persistent attacks in enterprise networks,” IEEE Transactions on Dependable and Secure Computing, 2023. [7] L. Nguyen, A. Zemmari, and H. Harroud, “Adaptive threat detection using dynamic graph learning for enterprise networks,” Journal of Network and Computer Applications, vol. 201, p. 103336, 2022. [8] Cisco Systems, “Snort - network intrusion detection & prevention system,” 2023, https://www.snort.org/. [9] OISF, “Suricata - open source ids/ips/nsm engine,” 2023, https://suricata. io/. [10] R. Dantu and P. Kolan, “Alert correlation engine for network security,” [11] Computer Communications, vol. 27, no. 15, pp. 1528–1535, 2004. [12] K. Julisch, “Clustering intrusion detection alarms to support root cause analysis,” ACM Transactions on Information and System Security (TIS- SEC), vol. 6, no. 4, pp. 443–471, 2003. [13] F. R. Kschischang, B. J. Frey, and H. A. Loeliger, “Factor graphs and the sum-product algorithm,” IEEE Transactions on Information Theory, vol. 47, no. 2, pp. 498–519, 2001. [14] D. A. R. P. Agency, “Darpa operationally transparent cyber dataset,” 2022, https://www.darpa.mil/program/operationally-transparent-cyber.

Copyright

Copyright © 2025 Mallipudi Beulah. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.